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ABSTRACT 

This report extends the results of an earlier study 
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accountability in future federal work force preparation legislation. 
It also illustrates specifically how the language of the Carl Perkins 
Vocational and Applied Technology Education Act of 1990 (Perkins II) 
could be changed to carry out the recommendations of tho earlier 
study. Four features are identified that were lacking in Perkins II 
but are important for an outcome-based system to promote effective 
program improvement: (1) coordinate separate components into a more 
integrated system for planning, implementing, monitoring, and 
improving vocational education and training; (2) increase the 
emphasis on the use of the system of performance measures and 
standards as a program improvement tool; (3) clarify the requirements 
for measures and standards and improve their technical quality; and 
(4) increase the amount of technical assistance provided by federal 
and state agencies to support change at the local and state levels. 
Specific examples are given of changes in the language of Perkins II 
to incorporate these principles. An appendix contains the complete 
text of the proposed revisions with detailed commentary comparing the 
new law to the old. (YLB) 
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PREFACE 



This report extends the results of an earlier NCRVE study, Improving Perkins II 
Performance Measures and Standards: Lessons Learned from Early Implementers in Four 
States, published m 1994, on the use of performance measures and standards to suggest 
principles for designing outcome-based program improvement. &ystems_iaiight of current 
efforts to reform the U.S. workforce education system. At the time this project was 
undertaken, states were actively involved in implementing the Carl D. Perkins Applied 
Technology and Vocational Education Act of 1990 (Perkins II), and educators and 
policymakers were anticipating the reauthorization of the legislation in 1995. However, the 
policy environment has changed; other options such as consolidated block grants for 
education and training are receiving increasing attention. Although in its details the present 
document reflects an emphasis on Perkins II, in its general principles the report should be 
of interest to federal policymakers engaged in developing new education and training policy 
initiatives. A companion RAND Issue Paper, Accountability and Workforce Training 
(Stecher & Hanser, 1995), discusses the implications of this study and related research on 
accountability in a non-Perkins environment. 
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SUMMARY 

The Carl D. Perkins Vcx:ational and Applied Technology Education Act of 1990 
(Perkins II) has guided federal vocational education policy for the past five years. One of 
the most significant components of Perkins II was its emphasis on using systematic 
outcome data as a program monitoring and improvement tool. A recent NCRVE study of 
the effects of Perkins IL Improving Perkins 11 Performance Measures and Standards: 
Lessons Learned from Early Implementers in Four States (Stecher et al., 1994), found that 
the performance measures and standards provisions designed to promote program 
improvement were not achieving their full potential; it identified shortcomings and 
recommended actions that could be taken to improve the act. 

This report examines the implications of that research for enhancing accountability 
in future federal workforce preparation legislation. It also illustrates specifically how the 
language of Perkins II could be changed to carry out the recommendations of the earlier 
study. 

The following four features were identified as lacking in Perkins U but were found 
to be important for an outcome-based system to promote effective program improvement: 

1. Coordinate separate components into a more integrated system for planning, 
implementing, monitoring, and improving vocational education and training. 

2. Increase the emphasis on the use of the system of performance measures and 
standards as a program improvement tool. 

3. Clarify the requirements for measures and standards and improve their technical 
quality. 

4. Increase the amount of technical assistance provided by state and federal agencies to 
support change at the local and state levels. 

Specific examples are given of changes in the language of Perkins II to incorporate 
these principles. 
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INTRODUCTION 

Current federal vocational education legislation expires in 1995, presenting ♦he 
104th Congress with an opportunity to reshape federal policy regarding secondary and 
postsecondary vocational education. At the time of this writing, it remains uncertain 
whether the Carl D, Perkins Vocational and Applied Technology Education Act (Perkins 
11) will be reauthorized in its present form or whether federal vocational education 
initiatives will be "merged into a broader workforce-development bill" (Sommerfeld, 
1994, p. 18). Given the growing emphasis on measurable outcomes and standards in 
government programs, requiremei^.ts for a system of outcome measures and standards are 
likely to be included in new legislation, regardless of its format. 

A recent study by the National Center for Research in Vocational Education 
(NCRVE), Improving Perkins II Performance Measures and Standards: Lessons 
Learned from Early Implementers in Four States (Stecher et al., 1994), suggests ways to 
enhance accountability in future federal vocational education legislation. This paper 
reviews the findings and recommendations of that study and illustrates how they could be 
translated into legislation. Specifically, we offer suggestions for more effective outcome - 
based program improvement and accountability procedures. The paper focuses only on 
those provisions that relate to evaluation and program improvement— often discussed 
under the heading "performance measures and standards/' These suggestions are 
presented as revisions to the existing Perkins legislation, but they are equally relevant in 
the context of broader workforce development legislation. Therefore, the report should 
be of general interest to policymakers responsible for workforce training at the state and 
federal levels. 

Examples of changes in language we suggest in this report are less relevant than 
tile principles that guided them. Specifically, the results of the study suggest that any 
federal vocational education effort should incorporate the following: 

• continued development of outcome-ba:^ed program assessment tools 

• greater coordination between vocational education and workforce development 
initiatives 
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• increased emphasis on the use of performance information for program 
improvement 

• increased technical assistance to support these objectives 

Furthermore, the language presented in this document may provide a useful starting point 
for other conceptualizations of thefederal role m vocational education. 

Background 

Perkins il has guided federal vocational education policy for the past five years. 
It has been a remarkably influential piece of legislation, in part, because most states have 
opted to apply its provisions more broadly than required. Although the federal 
government supplies less than ten percent of the total resources devoted to secondary and 
postsecondary vocational education, most states have applied Perkins II requirements to 
locally and state-funded vocational education efforts as well as to those funded with 
federal resources. 

One of the most significant changes embodied in Perkins 11 was an emphasis on 
usmg systematic outcome data as a monitoring and improvement tool for programs. 
Under Perkins EI, states were required to develop statewide **systems of core standards 
and measures of performance** that would be used to determine the success of vocational 
programs and to serve as a basis for local program improvement. If necessary, these 
systems could also be used to justify state intervention. States were given considerable 
flexibility in developing their statewide systems; the law specified only two outcomes 
that were required to be measured. At a minimum, each state was required to collect 
measures of learning and competency gains in basic and more advanced academic skills, 
as well as at least one measure of occupational competency attainment, job or work skill 
attainment, or retention or studcnt^laccrhcnt. Any additional measured outcomes were at 
the discretion of the Committees of Practitioners and state Departments of Education. 

Those who endorsed this approach to program monitoring in vocational education 
hoped it would lead to better evaluation, richer communication, more focused program 
improvement at the local level, and wiser use of state technical assistance capabilities. It 
was hoped that states would implement efficient systems that provided local 
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administrators and instructors with meaningful performance data to assess the strengths 
and weaknesses of their vocational programs and to design new strategies to improve the 
academic, technical, and labor-market outcomes of their students. 

These hopes have not been fully realized, ahhough states have made substantial 
strides in implementing performance measures and standards systems as envisioned in the 
federal legislation. However, there remains considerable room for improvement. The 
following summarizes the results of a recent NCRVE study on the implementation and 
impact of the Perkins II measures and standards. These findings form the basis for 
suggested changes in the legislation, which are elaborated in the section entitled 
"Rationale for Legislative Changes." 



NCRVE Study of the Implementation and Impact of Measures and Standards 

In the spring of 1993, NCRVE initiated a two-year study of the effects of Perkins 
II performance measures and standards for vocational education. At the time of the 
study, states had three years to implement thes^ provisions. The study examined seven 
states' progress in implementing statewide systems of pei^ormance measures and 
standards, the effects of these systems on local vocational programs and state agencies, 
and the factors that influenced local and state actions. ^ 

Four states which were "early adopters" of measures and standards v/ere initially 
selected for study. In each state, we interviewed staff in the state agency (or agencies) 
that administered se:.ondary and postsecondary vocational education, and administrators 
and instructors in both a secondary and a postsecondary vocational institution in two 
geographically separated regions. Respondents were asked about a number of related 
themes, including the implementation of performance measures and standards, their 
integration with other educational reform efforts, and the impact of measures and 
standards on their vocational programs. Repeat visits to each vocational institution were 
conducted the following year. In the second year, three additional states were added to 
the sample to provide greater contrast in terms of implementation and to broaden our 
exposure. 
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^ No formal attempt was made to detemiine whether states were in compliance with Perkins U or to judge 
Uie quality of die measures and standards states had chosen to implement 
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Results 

This section summarizes the findings of the NCRVE study (Stecher et al., 1994). 
Substantial progress had been made in implementing measures and standards in the states 
that were visited, although much work remains to be done to make the systems function 
as envisioned in the law. At ♦he time of our visits, little attention had been paid to 
building local- or state-level capacity for translating.the measures, and standards data into 
actions for local program improvement. These *Meading edge" states were still largely 
engaged in developing and implementing their systems. 

Furthermore, large variation was found in the states' approaches to the 
development and implementation of measures and standards. This variation was evident 
in almost every aspect of program implementation, including how the process was 
managed, who participated, and the level of resources devoted to it. These differences 
appeared to be jointly a function of the states* individuality and the flexibility inherent in 
Perkins II. 

We identified several factors that affected implementation and contributed to the 
variation in state responses to performance measures and standards. ^ Some of these 
explanatory factors were elements of the local and state context, and are less responsive 
to federal policy intervention. Other factors are within the sphere of federal policy 
influence. 

The level of flexibility afforded states was the first of five factors that could 
clearly be affected by federal actions. On the positive side, flexibility permitted states to 
create systems th?*^ ^ere responsive to local conditions. On the negative side, the latitude 
afforded states mcreased the influence of state context, which heightened differences 
between states, and, in some cases, lengthened the implementation process. 

The second explanatory factor was the separate and uncoordinated nature of the 
elements of Perkins U, The Perkins 11 priorities — measures and standards, integration, 
Tech Prep, and service to special populations — were not seen as a coordinated system at 
either the local or state levels. Similarly, performance measures and standards were not 
being used comprehensively to evaluate the other Perkins initiatives. 



^ See Stecher et at. (1994) for a complete description of the findings. 
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Third, there were neither models nor incentives for ensuring that performance 
measures and standards were used to improve programs. Perkins n contained an explicit 
framework fo/ j:truciuring systems, and a federal agency checked for compliance at the 
adoption stage. However, the law and regulations did little to emphasize information 
use. 3 

The fourth factor concerned resources and expertise at the state level. Perkins II 
created new responsibilities for state staff, but reduced the set-aside for state 
adnklnistration and provided little technical assistance. This presented a dilemma for 
states that lacked either the expertise or the resources to address these new demands. 

Finally, the law mandated measurement of learning outcomes, even though there 
were few valid tests available for this purpose. The scarcity of appropriate tools for 
measuring selected learning outcomes led states to adopt alternatives that were less than 
optimal. States are still struggling with how to measure important student outcomes such 
as academic skill gains at the postsecondary level. 

Recommendations 

The study recommended several actions federal policymakers could take to 
enhance the future success of performance measures and standards in vocational 
education and to promote the program improvement goals of Perkins II, 

• Clarify the Interrelationships and Coordination Among Federal Mandates. 

Policymakers should clarify the relationships among systems of measures and 
standards, the integration of vocational and academic education, Tech Prep 
programs, and service to special populations, as well as offer additional guidance 
about coordinating states* efforts in these areas. For example, performance 
measures and standards are clearly relevant to evaluating and improving School- 
to-Work transition- programs, and the links between these activities should be 
stressed in any future omnibus workforce legislation. 



3 Some sutc vocational educators told us that Perkins II could be improved by adding "real" incentives and 
sanctions, that is, by tying funding to program perforaiance. For example, they said that ''joint planning'* is 
not an adequate penalty for continued failure to make substantial progress toward meeting standards. 
Termination of funding might be more appropriate. This issue created controversy at the time the law was 
written, and it continues to generate debate today. This study did not provide adequate evidence to support 
or reject such a recommendation. 
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• Create Models for Outcome-Based Program Improvement 

At present, most state action is still driven by the mandate to develop the structure 
of measures and standards. Little has been done to use that structure to make 
programs better. Local and state agencies need assistance in translating outcome 
deficiencies into action plans. 

• Provide Focused Technical Assistance for Choices and Resources. 

The "flexible mandates" of Perkins II place greater demands on state agencies 
while restricting the use of funds for state-level services. Federal actions that help 
states respond to their choices and make better use of resources might 
significantly improve Perkins II implementation. States need resources and 
expertise to operate a reformed vocational education or workforce preparation 
system, and the law should promote both. 

• Address Conmion Measurement Problems. 

The technology to measure learning and occupational performance gains in 
reliable, valid, and efficient ways is not widely available. Most states are not 
equipped with either the resources or expertise to fill this gap. The federal 
government needs to assume leadership in addressing these problems, since they 
are best solved nationally and are largely the result of federal legislation 
provisions. 

Translating Results Into Legislation 

Incorporating these changes into the federal vocational education or workforce 
pre^raration legislation will increase the efficacy of statewide systems of measures and 
standards. While the results of this study do not speak to all aspects of the legislation, 
they do suggest to us specific changes in the measures and standards provisions. The 
section in this document entitled -"Rationale for Legislative Changes" presents our 
rationale for translating the findings of the study into changes in legislation. It elaborates 
on four main themes drawn from the results, and presents specific reconmiendations for 
revising the law. The next section, "Proposed 1995 Perkins Act Sections," contains a 
marked-up version of Perkins n showing where the changes might be made in the event 
of reauthorization. The paper concludes with a Technical Appendix that conU-asts the 
new and old language along with a commentary on proposed changes. 
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We remind the reader that we have not attempted to rewrite the whole act or to 
draft omnibus legislation. We aim only to demonstrate how results of our research have 
practical value in informing future legislation. Many of our suggested revisions to 
Perkins II could as easily be imbedded in a consolidated workforce preparation bill. 



The changes we propose flow directly from the research findings summarized in 
the previous section. In the case of performance measures and standards, the hnks 
between federal legislation and state actions are relatively clear and direct, and it is 
possible to trace implementation effects back to legislative causes. This linkage 
facilitates the task of revising the law to promote desired outcomes. We believe that the 
intent of the federal legislation was to promote effective program improvement, and that 
this can best be accomplished by including four major changes in future legislation: 

1. Coordinate separate components into a more integrated system for planning, 
implementing, monitoring, and improving vocational education and training. 

2. Increase the emphasis on the use of the system of performance measures and 
standards as a program improvement tool. 

3. Clarify and improve language describing the required measures and standards 
tliemselves. 

4. Increase the amount of technical assistance provided by state md federal agencies 
to support change at the local and state levels. 

To illustrate how these changes could be imbedded in federal legislation, we 
revised selected portions of* Sections- H5, 116, and 117 of Perkins U — the legislation 
describing performance measures and standards and the requirements for local and state 
assessment and evaluation. The proposed revisions have been italicized. Data limitations 
prevented us from undertaking a complete redrafting of the law. We have, therefore, 
limited our efforts to only those sections where our earlier findings justify legislative 
reworking. Readers iihould not assume that we endorse all the non-italicized sections of 
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the act; in most cases, these components are left unchanged because they were not 
informed by our research. 

The rest of this section describes in narrative format the major changes we 
recommend. This approach communicates better the goals we were trying to achieve and 
the broad changes we made to achieve them. The complete text of the proposed 
revisions, with detailed commentary comparing the new law to the old, is contained in the 
Technical Appendix. 

Developing a Coordinated Program Improvement System 

Our revisions attempt to coordinate the separate elements found in Perkins II into 
a more integrated system for planning, implementation, monitoring, and improving 
vocational programs. The logical model underlying this system is illustrated in Figure 1. 
Revised language clarifies the interrelationships between the elements of Perkins H. 
including state needs assessments, measures and standards, annual local evaluations, and 
program improvement plans. Our revisions also promote greater coordination of 
measures and standards with other federal workforce and education initiatives in the 
following ways: 

• Performance measures and standards are conceptualized as one part of a larger, 
interconnected system for improving vocational education at the local and state 
levels. The system includes many components, all of v*hich interact and inform 
one another. For example, the state needs assessment drives the system of 
measures and standards and the local application process. These, in turn, inform 
the annual local evaluation, which leads to program improvement planning, to 
periodic review of the system of measures and standards, and ultimately back to 
the state needs assessment. In this way the information comes full circle and 
helps to promote improvement of the whole system. 

• To make performance measures and standards a more comprehensive program 
improvement tool, they are applied to all major Perkins instructional initiatives 
(including programs funded under Title HI such as Tech Prep and consumer and 
home economics), not just to programs receiving funds under Title H. 
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The Committee of Practitioners is required to review and revise the measures and 
standards annually, creating a self-improving system that is more responsive to 
changes in other vocational education policies. 

Figure 1 

Proposed Legislative Relationships 



Secretary 
of 

Education 



State 
Board 



Cpmmtttet 

of 

Prmctthonert 



LOCAL 



Local 
Recipients 




Sec. 115 
Develop and implement z 
system for tmprovm;; 
vocational education 



Sec. 119 
Provtde techzucal 
asfistance 



Sec. 116 
Conduct a state needs 
assessment 


1 


f 


Revise cote 
measures ax 


117 

performance 
d standards 



Sute 
Plan 




Local 
Application 
Guideline* 




Local 
Application 




















Sec. 117 
Collect dau for 
performance measures 
and standards 



Sec. 118 
Annual evaluation 



See 118 
Develop tn annual 

local program 
improvement plan 



Sec US 
Develop >tate uvd local 
joint puuv if substantial 
prDgim is not made 



In our research, we recognized the clear need for coordination among federal 
workforce preparation programs. As a result, our suggested revisions in this area are 
particularly relevant in the context of a consolidated bill. Coordination among federal 
workforce education programs is promoted in the following ways: 

• The State Board and the Committee of Practitioners arc directed to consider all 
federal training and workforce preparation efforts when developing measures and 
standards and the program improvement system. Representatives of the agencies 
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responsible for administering related programs must be consulted in the 
appointment of the Committee of Practitioners. 

• The State Board is directed to provide the Committee of Practitioners with 
information about other state assessment efforts that might be relevant to a 
comprehensive program improvement system. 

• The federal government is directed to provide technical assistance to help states 
coordinate measures and standards with other federal initiatives. 

Use of Information for Program Improvement 

Our proposed changes represent a significant change in focus away from the 
initial development of the systems of measures and standards called for in Perkins II 
toward the use of these systems for future program improvement. Our research revealed 
that while states had made significant progress in developing their systems of measures 
and standards, for the most part they had not yet tackled the next step of using their 
systems for program improvement. Without explicit provisions for the use of 
performance measures and standards, the data they generate may languish in government 
files instead of being used to improve programs. Although the theme of measures and 
standards as the basis for a system of program improvement runs throughout our 
proposed revisions, it is most evident in Section 1 18, which sets specific requirements for 
program evaluation and improvement. 

Increased relevance and usefulness of the annual evaluations for program 
improvement is promoted in the following ways: 

• The requirement for program evaluation is explicitly written to include all the 
relevant component programs of Perkins (e.g.. Title in programs such as Tech 
Prep), not just those efforts funded under Title II. This change ensures that 
information about which components of Perkins are or are not working is 
available to assist in program improvement planning. 

• Teachers and parents are explicitly included in the evaluation process because 
they arc important stakeholders and can contribute to the improvement of 
programs« 
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H • The performance of special populations enrolled in vocational programs is to be 

specifically compared to the performance of other vocational students to ensure 

1^ that staff monitor the access and success of students from special populations in 

each program area. 

I 

• Additionally, program improvement activities are promoted in the following 

I ways: 

• The outcome of the annual evaluation is tied explicitly to a plan of program 
improvement emphasizing a link that is not sufficiently clear in current 
legislation. 

•To promote program improvement as a continuing and ongoing process, a 
requirement has been added for all recipients to develop a local improvement 
1^ plan, regardless of whether standards have been met. 

B* A requirement that the local improvement plan include a summary of the results 
of the local annual evaluauon strengthens and emphasizes the link between these 
activities. 

1 

•A requirement that those local recipients who fail to meet their standards describe 

• "specific strategies for making substantial progress" encourages follow-through 
fiom evaluation to program improvement. 

I 

Improved Measures and Standards 

I 

^ Measures and standards are the cornerstone of the Perkins program improvement 

• system. Our revisions attempt to improve measures and standards in two ways: (1) by 
clarifying the requirements for such systems and (2) by improving the technical quality of 
^ the components. Clarification is achieved by adding definitions, reformulating 

P descriptions, and reorganizing state op'iions to reveal more of the existing similarities. 

Technical quality is enhanced by requiring that states pay more attention to questions of 
^ reliability, validity, and lack of bias, and by requiring that the federal government 

contribute to the solution of common measurement problems. 

I 
I 
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Clarification of systems of measures and standards is achieved in the following 

ways: 

• Our revisions provide definitions for key terms that should be common 
throughout the country, including outcome, measure, standard, program, and 
special populations. Because of differing local contexts, we feel the definition of 
"substantial progress** ought to be a state prerogative, so the regulations require 
states to define this term in a measurable and consistent manner. 

• The requirements for measures and standards are reformulated and reorganized to 
make them more logical and consistent. State systems must contain at least five 
measures (most already contain more than this), one eacn from the following 
categories: learning gains, work-related skills, retention or completion, 
placement, and access/equity. This ensures that programs are evaluated based 
upon a complete and balanced picture of their desired effects. 

Technical quality is promoted in the following ways: 

• States are required to revise measures and standards regularly, based on an 
examination of technical quality and usefulness. This ensures that important 
policy decisions are based on sound information. 

• The federal government is required to contribute to the solution of common 
measurement problems such as evaluating the reliability, validity, and lack of bias 
in measures; measuring academic gains at the postsecondary level; and setting 
valid performance standards. These common problems, which were created by 
the legislation, are shared by all states and are most efficiently addressed 
nationally. 

• To focus attention on the goal of program improvement and the difficult problems 
of measures and standards; the Secretary is required to report to Congress on the 
status of each staters system for improving vocational education and on the 
technical quality of the measures and standards that arc adopted. 



ERIC 



NCRVE, MDS-821 



Technical Assistance 

Several activities required under Perkins IT were beyond the capacity of the local 
and state recipients to accomplish on their own. As a result, we have included a new 
Section 119, titled ^Technical Assistance/' that specifies actions the state and federal 
governments will take to assist recipients in carrying out their responsibilities for 
program improvement under reauthorized or consolidated legislation. We propose 
fundamentally different roles for the state and federal governments in the provision of 
technical assistance. Our distinction is based on those problems that we perceive as best 
solved nationally versus those that are primarily local and state in nature. The technical 
assistance role we propose for the federal government includes the following: 

• Assistance to help states address measurement-related issues, including 
establishing the technical quality of measures (i.e., reliability, validity, and lack of 
bias). This category also includes assistance in developing acceptable assessment 
tools. 

• Assistance with identifying, developing, and implementing program improvement 
strategies on a national level, in particular, helping to disseminate more broadly 
program improvement models and other lessons learned in individual states. 

• Assistance with coordinating the measures and standards requirements with other 
evolving and emerging workforce enhancement initiatives, including industry 
skill standards and School-to- Work performance standards. It is our belief that 
coordination will increase the effectiveness of these efforts. 

The states have the essential role of assisting local recipients in evaluating and 
improving their programs. This role includes the following responsibilities: 

• Assistance in evaluating the effectiveness of programs based on performance 
measures and standards* 

• Assistance in identifying and adopting appropriate strategies for improving 
performance. 

• Training local administrators and instructors in using performance data to 
improve vocational programs and courses. 
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Dissemination to local recipients of examples of effective performance oata use to 
improve vocational courses and programs. 



PROPOSED 1995 PERKINS ACT SECTIONS 

This section illustrates how the recommendations above could be translated into 
specific legislation. We present the reforms in the context of Perkins II because that was 
the relevant federal law at the time of this study. Similar wording would be appropriate 
for achieving similar goals in the context of a consolidated approach to workforce 
training. The changes are highlighted in italics in the following text. We did not attempt 
to rewrite the entire act, just to revise those sections that relate to program improvement 
and accountability. 

SEC. 115. IMPROVING VOCATIONAL EDUCATION AT THE STATE AND 
LOCAL LEVEL 

(a) GENERAL AUTHORITY— Each State Board receiving funds under this Act 
shall develop and implement a system for improving vocational education at the 
State and local level that includes: 

(1) reappointing a State Committee of Practitioners (Section 1 15); 

(2) conducting a State needs assessment in the first year under 
reauthorization and every two years thereafter ( Section 1 16); 

(3) implementing a set of core performance measures and standards (Section 
117); 

(4) conducting annual evaluations of program effectiveness (Section 118); 
and 

(5) providing technical assistance to local recipients on using performance 
data to improve vocational courses and programs (Section 119). 

This system shall build on the system initiated under the 1990 Perkins Act, and 
apply to all programs (as defined in Subsection (c)) receiving funds under Title 11 
and as appropriate, to programs funded under Title III of this Act. 

(b) REQUIREMENT— Each State Board, before the expiration of the 30-day period 
beginning on the date of the enactment of the 1995 Perkins Act, shall reappoint a 
^tatc Committee of Practitioners (referred to as the "Committee'*) to develop the 
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elements of a program improvement system contained under the above general 
authority. Committee members will be appointed in consultation with local 
school officials representing eligible recipients, and representatives of org:inized 
labor, business, superintendents, community-based organizations, private industry 
councils established under Section 102(a) of the Job Trannng PaitnersLip Act, 
State councils, parents, special populations, correctional institutions, the 
administrator appointed under Section 111(b)(1), the State administrator of 
programs assisted under Part B of the Education of the Handicapped Act, the 
State administrator of programs assisted jnder Chap^.er 1 of Title I of the 
Elementary and Secondary Education Act, the State administrator of programs for 
students of limited English proficiency, and guidance counselors, and 
additionally — the State administrator of programs developed under Title //. 
Subtitle A of the School-to-Work Opportunities Act of 1994, and the State 
administrator of programs assisted mider Title HI of the Goals 2000: Educate 
America Act. 

DEFINITIONS— The State Board shall adopt the following definitions in its 
State plan: 

(1) Outcome — a measurable characteristic of student or program 
performance; 

(2) Measure — a method for quantifying performance in an outcome area; 

(3) Standard— the desired level of attainment on a measure, or the rate of 
progress on a measure; 

(4) Program — a sequence of courses or instruction in a sequence or 
aggregation of occupational competencies that includes occupationally 
specific skills, general employability skills, work attitudes, applied 
academic learning, higher-order reasoning, and problem-solving skills; 

(5) Special populations— includes individuals with disabilities, educationally 
and economically disadvantaged individuals (including foster children), 
individuals of limited English proficiency, individuals in programs 
nontraditional for their sex, and individuals in correctional institutions. 

The State Board shall also define the term "substantial progress*' in its State plan 
so that the term is measurable and applies consistently to all recipients of Perkins 
funds in the state. 
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SEC- 116. STATE NEEDS ASSESSMENT 

(a) IN GENERAL — Each State Board receiving assistance under this Act shall 
conduct an assessment using measurable objective criteria developed by tne State 
Board to assess program quality. Such criteria shall be developed in consultation 
^ith representatives of the Committee described in Section 1 15(^7) and shall use 
information gathered by the National Occupational Information Coordinating 
Committee and, as appropriate, other information. Each State Board shall widely 
disseminate such criteria. State Boards shall develop such criteria no later than 
the beginning of the 1 996- 1 997 school year. Such criteria shall include — 
(!) Information on outcomes — 

(A) student gains in and/or attainment of academic, occupational, 
and/or generic work skills: 

(B) student retention in and/or completion of secondary or 
postsecondary education and/or vocational programs: 

(C) student placement into additional training or education, military 
service, and employment: 

(D) other outcome areas identified by the State under Section 115 of 
the 1990 Perkins Act; 

(2) Information on instructional practices and experiences — 

(A) integration of academic and vocational education; 

(B) the ability of the eligible recipients to meet the needs of special 
populations with respect to vocational education, including 
providing incentives for participation, retention, and completion; 

(C) assessing the quality of vocational education programs in schools 
with high concentrations of poor or low-achieving students; 

(D) enrollment retention, and completion rates of students in 
vocational programs nontraditional for their sex; 

(E) sequential courses of study leading to both academic and 
occupational competencies; 

(F) linkages between secondary and postsecondary educational 
institutions; 

(G) instruction and experience, to the extent practicable, in all aspects 
of the industry the students are preparing to enter, 

(H) the relevance of programs to the workplace ana to the occupations 
for which students are to be trained, and the extent to which such 
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programs reflect a realistic assessment of current and future labor 

market needs, including needs in areas of emerging technologies; 
(I) the ability of the vocational curriculum, equipment, and 

instructional materials to meet the demands of the workplace; 
(J) basic and higher order current iiiid future workplace competencies 

that will reflect the hiring needs of employers; 
(K) other factors considered appropriate by the State Board; and 
(3) Procedures for using existing: resources and methods developed in other 
programs receiving Federal assistance. 

(b) DEADLINE FOR ASSESSMENT— Each State Board shall complete the 
assessment required by Subsection (a) before the expiration of the 6-month period 
beginning on the date of the enactment of the 1995 Perkins Act, and every two 
years thereafter. 

SEC. 117. STATE AND LOCAL IViEASURES AND STANDARDS 
(a) GENERAL AUTHORITY— Each State Board receiving funds under this Act 
shall revise and implement a set of core standards and measures of performance 
for secondary and postsecondary vocational education programs. To advise the 
State Board, the Committee shall— 

(1) evaluate the reliability, validity, and lack of bias of the measures and 
standards developed under Section 115 of the 1990 Perkins Act; 

(2) assess the extent to which data generated under Section 115 of the 1990 
Perkins Act were used effectively by recipients to improve programs and 
identify barriers to effective use; 

(3) recommend revisions to existing measures and standards; and 

(4) develop new measures and standards as required under Subsection (b). 
Revisions to the core measures and standards will be recommended by the 
Committee and approved by each State Board no later them one year after the 
enactment of the 1995 Perkins Act. This core shall apply to all programs (as 
defined in Section 11 5(c)(4)) receiving funds under Title U of this Act. 
Additiormlly, the State Board may require that the core apply to any programs 
funded under Title HI such as consumer and homemaking education and tech- 
prep. Eligible recipients may make local modifications to such core based on 
economic, geographic, or demographic factors, or the characteristics of the 
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population to be served. Such modifications shall conform to the assessment 
criteria coiitaincd in the State plan. The State Board shall convene the Committee 
on a regular basis to review, comment on, and propose revisions to the State's 
core standards and measures of performance for vocational programs, 

(b) REQUIREMENTS— Each system developed under Subsection (a) shall include 
at least five measures and standards — 

(1) a measure of student learning gains in the achievement of academic skills: 

(2) 1 or more of the following measures of work-related skills — 

(A) occupational competency attainment in the industry a student is 
preparing to enter; 

(B) attainment of work readiness or generic work skills; 

(C) attainment of understanding of all aspects of the industry a student 
is preparing to enter; 

(3) 1 or both of the following measures of retention or completion — 

(A) retention in or completion of secondary or postsecondary 
education; 

(B) retention in or completion of secondary or postsecondary 
occupational programs or other programs funded under Title III, 
as appropriate; 

(4) a measure of placement into additional training or education (including 
placement into the postsecondary component of tech-prep or other School- 
to-Work transition program), military service, or employment: and 

(5) a measure of access and equity such as rates of enrollment in, retention in, 
and/or completion of occupational or other programs for special 
populations in comparison to non-special populations, including male and 
female students in programs nontraditionalfor their sex. 

(c) CONSISTENCY AND COORDINATION WITH OTHER PROGRAMS— In 
developing the standards and measures included in a system developed under 
Subsection (a), the State Board shall — 

(1) consider standards and measures developed under job opportunities and 
basic skills training programs established and operated under a plan 
approved by the Secretary of Health and Human Services that meets the 
requirements of Section 402(a)(19) of the Social Security Act; 
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(2) consider standards prescribed by the Secretary of Labor under Section 106 
of the Job Training Partnership Act; 

(3) coordinate with State efforts to implement performance measures 
developed under Title IV, Section 402 of the School-to-Work Opportunities 
Act; 

(4) build on the content and student, performance, standards and related 
assessments developed under Title 11 Part B of the Goals 2000: Educate 
America Act in assessing student learning gains in the achievement of 
academic skills at the secondary level; and 

(5) build on the industry skill standards and related assessments and 
certifications developed under Title V of the Goals 2000: Educate 
America Act (the National Skill Standards Act) in assessing student 
attainment of occupational competencies at the secondary level, and at the 
postsecondary level as appropriate. 

(d) INFORMATION PROVIDED BY STATE BOARD— 

(1) The Committee shall make recommendations to the State Board with 
respect to modifying standards and measures to be used under this section, 
based on the information provided under paragraph (2) and on the state 
needs assessment described under Section 118. 

(2) In order to improve the system of measures and standards as directed 
under Subsection (a), the State Board shall provide to the Committee 
information concerning differing types of standards and measures, 
including — 

(A) the advantages and disadvantages of each type of standard or 
measure; 

(B) instances in which such standards and measures have been 
effective; 

(C) instances in which such standards and measures have not been 
effective; 

(D) information about standards, measures, and assessment 
instruments already in use statewide that may be incorporated into 
the core set of performance measures and standards; and 

(E) information provided through federal technical assistance as 
described under Section 119, as appropriate. 
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(3) In the event that the State Board does not accept the Committee's 
recommendations made as required by paragraph (1), the State Board shal! 
set forth in the State plan its reasons for not accepting such 
recommendations. 

(e) SUBMISSIONS— Each State Board will submit to the Department of Education 
as part of its State plan: ( I) the measures and standards to be used, and (2) an 
implementation plan including plans for providing technical assistance to local 
recipients of Perkins funds, as required under Section 1 19. 

SEC 118. PROGRAM EVALUATION AND IMPROVEMENT 
(i^) ANNUAL EVALUATION — Each recipient of financial assistance under Part C 
of Title II shall annually evaluate the effectiveness of the programs conducted 
with assistance under this Act, including programs funded under Title III as 
determined by the State Board in Section 117(a). This evaluation shall he 
accomplished in consultation with teachers, parents, representatives of 
individuals who are members of special populations, and industry representatives. 
As part of such evaluation, each such recipient shall — 

(1) evaluate the effectiveness of programs with respect to performance on 
each measure and standard developed under Section 1 1 7; 

(2) evaluate the performance of special populations in comparison to non- 
special populations on each such measure and standard; 

(3) evaluate the contribution of the instructional practices and experiences 
identified under Section II 6(a)(2) to performance on the measures and 
standards; 

(4) evaluate the contribution of any projects, services, and activities 
conducted with assistance under this Act to performance on the measures 
and standards; and 

(5) identify and adopt strategies to improve performance on the measures and 
standards. 

(b) LOCAL PROGRAM IMPROVEMENT PLAN— Beginning not less than 1 
year after the implementation of the provisions of Section 117, all recipients of 
funds under Part C of Title II will develop a local program improvement plan that 
summarizes the self-evaluation conducted under Subsection (a). A recipient that 
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does not meet the standards under Section 117 must describe in its local program 
improvement plan specific strategies for making substantial progress (as defined 
by the State Board under Section 115(c)) toward those standards in the coming 
year Such local program improvement plans shall be updated annually, 

(c) STATE AND LOCAL JOINT PLAN— If, L year after impl^nientation of the 
plan described in Subsection (b), and each year thereafter, substantial progress in 
meeting the standards and measures developed under Section 117 has not been 
made, the State shall work jointly with the recipient, and teachers, parents, and 
students concerned, to develop a plan for program improvement. Each such plan 
shall contain — 

(1) a description of the technical assistance and program activities the State 
will provide to enhance the performance of the recipient; 

(2) a reasonable timetable to improve recipient performance under the plan; 

(3) a description of vocational education strategies the recipient will adopt to 
improve performance on the measures: and 

(4) if necessary, a description of strategies the recipient will adopt to improve 
supplementary services provided to individuals who are members of 
special populations. 



(d) FURTHER ACTION— The State shall, in conjunction with the eligible 
recipient, annually review and revise the joint plan developed under Subsection 
(c) in order to improve program performance and will continue to do so each 
consecutive year until the recipient sustains, for more than 1 year, substantial 
progress toward or meets the standards developed under Section 117. 

SECTION 119. TECHNICAL ASSISTANCE 

(a) FEDERAL ASSISTANCE— The Secretary shall provide technical assistance to 
the States, directly or by grant; contract; or cooperative agreement, with respect 
to the development and implementation of systems under Section 115 and 
subsequent sections, including assistance on — 

setting performance standards; 
(2) evaluating the reliability, validity, and lack of bias of measures and 
standards; 
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(3) selecting and developing appropriate assessment tools, including methods 
for assessing academic gains at the postsecondary level; 

(4) coordinating Perkins measures and standards with other evolving and 
emerging federal initiatives such as National Industry Skill Standards and 
School-tO'Work performance measures; 

(5) collecting and disseminating information on models for using performance 
data for program improvement, particularly at the local level; and 

(6) planning for and providing technical assistance to local recipients. 

In providing such assistance, the Secretary shall utilize existing resources in other 
Federal agencies. 

(b) STATE ASSISTANCE— Each State Board shall provide technical assistance, 
directly or by grant, contract, or cooperative agreement, to local recipients with 
respect to the implementation of Sections 117 and 118, including — 

(1) assistance on evaluating the effectiveness of programs based on the 
performance measures and standards developed under Section 1 1 7; 

(2) assistance on identifying and adopting appropriate strategies for 
improving performance on such measures and standards; 

(3) training local administrators and instructors in the use of performance 
data to improve vocational courses and programs based on the assessment 
conducted under Section 117(a); and 

(4) disseminating to local recipients examples of the effective use of 
performance data to improve vocational courses and programs. 

Section 102 allocates an annual amount for the technical assistance activities 
idmtified under this subsection from the allotment made to each State from funds 
appropriated under Section 3(al 

SEC. 120. EVALUATING STATE AND NATIONAL SYSTEMS 
(a) REPORT — ^The Secretary shall submit a report to the appropriate committees of 
the Congress not later than the expiration of the 4-year period beginning on the 
date of the enactment of the Carl D. Perkins Vocational and Applied Technology 
Education Act Amendments of 1995. Such report shall include- 
(1) a detailed description of the status of each State's system for improving 
vocational education at the State and local level developed as required by 
Section 115 and subsequent sections; 
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H (2) an assessment of the reliability, validity, and lack of bias of the measures 

and standards adopted under Section 1 1 7; and 
m (3) an assessment of the degree to which the use of these systems haye led to 

program improvement at the State and local levels. 
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